PCI-Express (PCIe) as the successor of the PCI (Peripheral Component Interconnect) technology is the most widely used interconnect between a central processing unit (CPU) and its peripherals, as deployed within PCs and servers. PCIe provides for a high throughput, low-latency, packet based and switched interconnection technology. PCIe is currently mostly deployed within single enclosures (i.e., servers and PCs) and primarily at the printed circuit board (PCB) level. PCIe with all its attributes and advantages, can clearly become a flexible and cost efficient alternative to traditional Data Center interconnect technologies, such as Ethernet (ETH) and InfiniBand (IB).
For instance, a single PCI-Express link can scale up to 256 Gbps (gigabits-per-second) with a latency of approximately 130 nanoseconds per switch hop. The bandwidth of links can be flexibly configured from 1 to 16 lanes, wherein lane counts have to be a power of 2, with 8 Gbps per lane (as of Generation 3 PCIe technology). PCIe switches are capable of interconnecting links with different speeds (i.e. number of lanes) and lane counts per link (i.e. switch port) can be dynamically reconfigured.
The PCIe technology provides hardware level resource sharing with a large set of different PCIe based I/O devices supporting this technology. The technology itself is so widely used within servers, that the majority of the latest CPUs already have PCIe built into the chip, not requiring any additional chipset to be connected between the CPU and the peripherals. With this, clusters or Data Centers based on a PCIe interconnect, would achieve greatly improved cost efficiency, due to the lower number and types of components needed in the system. In addition, compared to ETH and IB the power consumption of PCIe switches is several times lower per Gbps, further contributing to the cost efficiency of such a system.
As per functionality, besides hardware-level resource virtualization, it also provides RDMA (Remote Direct Memory Access) functionality, making it a viable solution for high-speed, low-latency CPU-to-CPU communication. PCIe also uses built in credit-based flow-control, providing reliable data transfer at the hardware level, requiring a much “slimmer” software stack. The technology also supports cabling for the network to span larger distances and is flexible enough to support different network topologies. There are several well-proven solutions with low cost adapters and cables, making it a potential, cost-efficient Data Center interconnect.
Another great advantage of the PCIe technology is that its standardization organization (i.e. PCI-SIG) only defines the protocol and packet formats. That is, vendors implementing PCIe are free to implement any functionality inside their devices (i.e. switches, end-points, etc.). As such, PCIe provides a great level of flexibility and potential for supporting different functionalities for applications missing from other Data Center interconnection technologies.
In particular, the PCIe architecture is a point-to-point topology, with serial links connecting every device to the root host. PCIe devices communication via logical interconnections or links. The link provides for point-to-point communications over a channel between two PCIe ports. This allows both ends to send and/or receive ordinary PCIe requests (e.g., configuration read/write, I/O read/write, memory read/write, etc.), and interrupts (INTx, MSI, MSI-X, etc.).
In a typical use of PCIe of communication between a CPU and its peripherals, initialization of an interrupt provides for programming a memory address as a destination address (e.g., of an interrupt handler) at the corresponding peripheral. Initialization allows for delivery of interrupts from the peripheral to the CPU for handling. When everything is internal to the CPU, the CPU can properly program its peripherals during interrupt initialization. However, when extending PCIe for use as a unified interconnect technology between components of a data center, interrupt initialization is problematic because the CPU programming the interrupt at the device may not be aware of the proper address. In that case, initialization will fail.
It would be advantageous to provide for interrupt initialization in off-the-shelf PCIe devices for multiple independent CPUs when PCIe is used as a interconnect technology between components of a Data Center.